Asynchronous Distributed Data Parallelism for Machine Learning

Distributed machine learning has gained much attention due to recent proliferation of large scale learning problems. Designing a high-performance framework poses many challenges and opportunities for system engineers. This paper presents a novel architecture for solving distributed learning problems in framework of data parallelism where model replicas are trained over multiple worker nodes. Worker nodes are grouped into worker groups which enable model replicas to be asynchronously aggregated via peer-to-peer communication. Merits of this framework include elastic scalability, fault tolerance, and efficient communication.